Real-time typing assistance

ABSTRACT

An apparatus and method are disclosed for providing feedback and guidance to touch screen device users to improve the text entry user experience and performance through the use of indicators such as feedback semaphores. Also disclosed are suggestion candidates, which allow a user to quickly select next words to add to text input data, or replacement words for words that have been designated as incorrect. According to one embodiment, a method comprises receiving text input data, providing an indicator for possible correction of the text input data, displaying suggestion candidates associated with alternative words for the data, receiving a single touch screen input selecting one of the suggestion candidates, and modifying the input data using the word associated with the selected suggestion candidate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/304,341, filed Feb. 12, 2010, and entitled “TYPING ASSISTANCE FOREDITING,” which is hereby incorporated herein by reference in itsentirety.

FIELD

The present disclosure pertains to devices and methods for enhancingtext entry using a touch screen device.

BACKGROUND

With the increasing popularity of mobile devices, including cellphonedevices, handheld devices, handheld computers, smartphones, PDAs, etc.,there is a need for improving the user interface experience byincreasing user text input speed, reducing text entry errors, andimproving the overall user experience.

Mobile devices with capacitive or resistive touch capabilities oftenutilize a touch screen keyboard, a hardware keyboard, speechrecognition, handwriting recognition, or combination of the four, forentry of text input. Touch screen keyboards enable larger displays forvideos, web pages, email, etc., without the requirement of a physicalkeyboard. Because touch screen keyboards are software-based, they can beeasily adjusted for different languages, screen orientation, and keylayouts. Furthermore, touch screen keyboards can be augmented withwidgets for word prediction and disambiguation candidates.

Users of devices with touch screens, especially mobile devices, havevarying abilities and styles of entering text. In particular, some usersprefer to type large chunks of text input fairly rapidly, and do not toreview and correct the entered text until complete phrases, sentences,or complete messages have been entered. Similarly, users entering textusing speech recognition or handwriting recognition do not want to stopto review their text input until having entered completed phrases,sentences, or complete messages. Predictive typing assistance softwaresuch as T9 only offer word prediction candidates as users type. Afterusers finish typing, they are usually left without any assistance, andmust then struggle to edit text by placing cursors in-between charactersin order to proof and correct text.

Therefore, there exists ample opportunity for improvement intechnologies related to facilitating user input on electronic devices byproviding more helpful and accurate assistance in the text correctionprocess in order to accelerate user text entry and reduce user inputerror rates.

SUMMARY

An apparatus and method are disclosed for providing feedback andguidance to touch screen device users to improve the text entry userexperience and performance.

One exemplary embodiment disclosed herein is a method comprisingreceiving first input data comprising one or more input words from akeyboard, providing one or more feedback semaphores, wherein thefeedback semaphores are operable to alert a user using the keyboard thatone or more suggestion candidates are available, automaticallydisplaying the one or more suggestion candidates, wherein the suggestioncandidates are each associated with an alternative word for one or moreof the input words, automatically receiving a touch screen inputselecting one of the suggestion candidates, and modifying the firstinput data using the alternative word associated with the selectedsuggestion candidate. In some examples, one or more alternative wordsfor one of the input words are generated, the first input data isimmediately auto-corrected are providing the feedback semaphores byautomatically replacing the input word with one of the alternativewords. In some examples, the first input data comprises a characterdesignated as a delimiter and the providing feedback semaphores occursbased on a word probability for the input word and receiving thedelimiter character. In some examples, the feedback semaphores caninclude one or more of the following: playing a sound, generating hapticfeedback, highlighting one or more keys on the keyboard, highlighting abackground area of the keyboard, and highlighting a suggestioncandidates area. In some examples, the feedback semaphores includehighlighting a delimiter key and receiving a single touch screen inputfor one of the highlighted delimiter keys. In some examples, thefeedback semaphores are not provided until the input word is designatedincorrect using a candidate source. In some examples, the feedbacksemaphores provided can be selected based on receiving typing speed datafor the first input data. In some examples, the feedback semaphoresprovided can be selected based on detecting whether the user is typingon the keyboard with one or two hands.

Another exemplary embodiment disclosed herein is a computer-readablestorage media storing computer-readable instruction that when executedby a computer cause the computer to perform a method comprising:receiving text input data comprising at least one word using a keyboardcoupled to a touch screen, detecting that an event has occurred based onthe text input data and based on the detected event, automaticallydisplaying on the touch screen one or more suggestion candidates for theat least one word based on the text input data and one or more candidatesources, where each of the candidates is associated with at least onenext word designated as likely to follow the at least one word,receiving a single touch screen input selecting one of the suggestioncandidates, and, based on the single touch screen input, automaticallymodifying the text input data by adding the at least one next wordassociated with the selected suggestion candidate to the text inputdata. In some examples, the event is detected based on a measuredkeystroke latency. In some examples, the event is an auto-correction ofthe text input data. In some examples, the event is receiving adelimiter key in the text input data. In some examples, a method furthercomprises automatically displaying on the touch screen one or morereplacement suggestion candidates for the at least one word based on thetext input data and one or more candidate sources, where each of thereplacement suggestion candidates is associated with at least onereplacement word for the at least one word, and the displaying includesdisplaying the replacement suggestion candidates in a manner thatdistinguishes the replacement suggestion candidates from the suggestioncandidates. In some examples, a method comprises immediately after theautomatically adding the at least one word, displaying one or moresuggestion candidates for the text input data, which includes theselected next word, and repeating the automatically modifying the textinput data.

A further exemplary embodiment disclosed herein is a mobile devicecomprising one or more processing units operable to executecomputer-executable instructions for text entry and correction, one ormore memory units coupled to the processing units, one or more touchscreens coupled to the mobile device configurable to have a text displayarea, a suggestion candidates area, and a touch screen keyboard area,wherein the text display area, the suggestion candidates area, and thetouch screen keyboard area occupy distinct, non-overlapping areas of thetouch screens, and wherein the one or more touch screens are operable toreceive touch input over at least a portion of the touch screen keyboardarea and the suggestion candidates area. A mobile device furthercomprises storage for storing the computer-executable instructions fortext entry and correction using a text input module for receiving textinput using the touch screen keyboard and displaying at least a portionof the text input in the text display area a candidate generation modulefor generating one or more suggestion candidates comprising one or morereplacement suggestions and/or one or more next word suggestions for aninput word of the text input an indicator generation module forproducing one or more indicators that can notify a user that thesuggestion candidates are available, a suggestion presentation modulefor presenting the suggestion candidates associated with the input wordin the suggestion candidates area and receiving touch screen user inputfor selecting one of the presented suggestions wherein if the selectedsuggestion is a replacement suggestion, replacing the input word withthe selected suggestion, and if the selected suggestion is a next wordsuggestion, adding the selected suggestion as a next word subsequent tothe input word in the text input. In some examples, the indicatorgeneration module and the suggestion presentation module are configuredto not produce indicators and to not present suggestion candidates,respectively, until one of the input words is designated incorrect by adictionary, a common speller application programming interface, or aninput history data source. In some examples, the indicator generationmodule and the suggestion presentation module are configured to notproduce indicators and to not present suggestion candidates,respectively, until a delimiter character is received immediately afterthe input word.

The described techniques and tools for solutions for improving textentry user experience and performance can be implemented separately, orin various combinations with each other. As will be described more fullybelow, the described techniques and tools can be implemented on hardwarethat includes software touch screen keyboards or hardware keyboards. Aswill be readily apparent to one of ordinary skill in the art, thedisclosed technology can be implemented using various platforms coupledwith a touch screen including, but not limited to, mobile devices(cellphones, smartphones, PDAs, handheld devices, handheld computers,PDAs, touch screen tablet devices), tablet or laptop computers, desktopcomputers, and home theater systems. As used herein, a touch screenincludes a display coupled with touch sense capabilities (for example,displays using capacitive or resistive sensors).

The foregoing and other objects, features, and advantages will becomemore apparent from the following detailed description, which proceedswith reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system diagram depicting an exemplary mobile deviceincluding a variety of optional hardware and software components.

FIG. 2 illustrates a generalized example of a suitable computingenvironment in which described embodiments, techniques, and technologiesmay be implemented.

FIG. 3 illustrates a generalized example of a suitable implementationenvironment for a text entry device connected to a computing cloud.

FIG. 4 illustrates a generalized example of a text entry device having atouch screen including a touch screen keyboard, a suggestion candidatesarea, and a text entry area.

FIGS. 5A-5C illustrate generalized examples of text entry devices,including a touch screen device with a touch screen keyboard, a mobiledevice, and a personal computer.

FIG. 6 is a flow chart of an exemplary implementation of a method forgenerating possible corrections for input data, providing indicatorsincluding feedback semaphores that indicate the availability of thepossible corrections, and displaying suggestion candidates.

FIGS. 7A-7E depict a method of providing unexpected-key feedback,including providing feedback semaphores, when an unexpected key ispressed.

FIGS. 7F-7H depict three alternative designs for providing feedbacksemaphores.

FIG. 8 is a flow chart that outlines an exemplary implementation of themethod shown in FIGS. 7A-7E.

FIGS. 9A-9D depict an exemplary implementation of a method of providingauto-correction notification, including providing feedback semaphores,when auto-correction of text input data occurs.

FIG. 10 is a flow chart that outlines an exemplary implementation of themethod shown in FIGS. 9A-9D.

FIGS. 11A-11C depict an exemplary implementation of a method ofproviding likely next phrases, including using suggestion candidatesafter receiving a text input word and a delimiter.

FIGS. 11D-11F depict an exemplary implementation of a method ofproviding likely next words or phrases, including using suggestioncandidates after receiving a text input word and a delimiter, and thenproviding an additional next word.

FIG. 12 is a flow chart that outlines an exemplary implementation of themethods shown in FIGS. 11A-11C and 11D-11F.

FIG. 13 is a chart representing events that occur during an exemplaryimplementation of a method that includes playing random click sounds asa user presses keys on a touch screen keyboard, as well as the use ofkeystroke latency to determine when to present suggestion candidates.

FIG. 14 is a flow chart that outlines an exemplary implementation of themethods shown in FIG. 13.

FIG. 15 illustrates a generalized example of a suitable implementationenvironment including a computing cloud and various connected devices inwhich described embodiments, techniques, and technologies can beimplemented.

FIGS. 16A-16D are charts depicting some experimental results obtainedusing the disclosed technologies.

DETAILED DESCRIPTION I. General Considerations

This disclosure is set forth in the context of representativeembodiments that are not intended to be limiting in any way.

As used in this application and in the claims, the singular forms “a,”“an,” and “the” include the plural forms unless the context clearlydictates otherwise. Additionally, the term “includes” means “comprises.”Further, the term “coupled” encompasses mechanical, electrical, as wellas other practical ways of coupling or linking items together, and doesnot exclude the presence of intermediate elements between the coupleditems.

The described things and methods described herein should not beconstrued as being limiting in any way. Instead, this disclosure isdirected toward all novel and non-obvious features and aspects of thevarious disclosed embodiments, alone and in various combinations andsub-combinations with one another. The disclosed systems, methods, andapparatus are not limited to any specific aspect or feature orcombinations thereof, nor do the disclosed things and methods requirethat any one or more specific advantages be present or problems besolved.

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed things and methods can be used in conjunction with otherthings and methods. Additionally, the description sometimes uses termslike “produce,” “generate,” “select,” “highlight,” and “provide” todescribe the disclosed methods. These terms are high-level abstractionsof the actual operations that are performed. The actual operations thatcorrespond to these terms will vary depending on the particularimplementation and are readily discernible by one of ordinary skill inthe art.

Theories of operation, scientific principles or other theoreticaldescriptions presented herein in reference to the apparatus or methodsof this disclosure have been provided for the purposes of betterunderstanding and are not intended to be limiting in scope. Theapparatus and methods in the appended claims are not limited to thoseapparatus and methods that function in the manner described by suchtheories of operation.

In the following description, certain terms may be used such as “up,”“down,” “upper,” “lower,” “horizontal,” “vertical,” “left,” “right,”“over,” “on,” “near,” and the like. These terms are used, whereapplicable, to provide some clarity of description when dealing withrelative relationships. But, these terms are not intended to implyabsolute relationships, positions, and/or orientations.

As used in this disclosure, the term “wait” may be used to describe theaction a device takes while waiting for particular value or type ofinput before proceeding with a particular operation. This waiting shouldnot be construed as limiting the device to only waiting for theparticular type of input, rather, the device may receive other input orperform other actions concurrently with the waiting.

As used in this disclosure, the term “automatically” is used to describeactions that can proceed immediately, without receiving further userinput. As used in this disclosure, the term “immediately” means that anaction occurs within a short time period following a preceding actionwithout needing to receive intervening user input. In some cases, theremay be intervening actions performed between or concurrently with thepreceding action and the action occurring “immediately,” for example,screen refresh or redraw, sound playback, etc.

As used in this disclosure, the term “incorrect” is used to describe adesignation of a word or phrase as being incorrect. A word designated asincorrect can be automatically highlighted or auto-corrected, eventhough the word designated as incorrect by a correction module mightactually be considered to be correct by the user. For example, a wordcan be designated as incorrect because it does not exist in adictionary, CSAPI (common speller application programming interface), orIHDS (input history data source). Alternatively, a word can bedesignated as “incorrect” even though it exists in a dictionary, CSAPI,or IHDS, because of other checking rules implemented in a correctionmodule or candidate generation module, or because of the context of theword within a phrase.

As used in this disclosure, the term “indicator” is used to describeoutput that is intended to capture the attention of a user of a textentry device, and can include visual, audio, and haptic feedback. Someexamples techniques for providing visual indicators include highlightingan object on a display by changing the color of the object, shading ofthe object or background of an object, changing the color of one or moreelements of an object (e.g., by coloring the letter “F” on the “F” keyof a touch screen keyboard), flashing the display, or by drawing a newobject (e.g., using underlines, balloons, or suggestion candidates) onthe display. Some examples of audio indicators include playing back apre-recorded or synthesized sound, including a randomly selected sound.Some examples of haptic indicators include vibrating a handheld deviceor area of a touch screen keyboard, as well as increasing or decreasingthe amount of pressure sufficient to register a key press on a keyboard,including touch screen keyboards. An example of varying key pressresistance to prevent typing errors may be found in Hoffmann, et al.,“TypeRight: A Keyboard with Tactile Error Prevention,” Proceedings ofthe 27th Int'l Conf. on Human Factors in Computing Systems, pp.2265-2268 (2009).

As used in this disclosure, the term “semaphore” is used to describe theuse of one or more indicators to provide feedback and/or guidancesignals on a text entry device user to help discover and avoid textentry errors. Generally speaking, “feedback semaphore” is used todescribe an indicator or signal that is provided after an event isreceived (for example, an event could be an auto-correction or otheraction that a text entry device performs, or the receiving of a keypress or touch screen touch input). In contrast, “guidance signal” isused to describe an indicator that is provided before the occurrence ofan event. For example, a text entry device can predict which key a useris likely to press next and highlight the most likely key, therebyproviding a guidance signal, so the user can more easily find the mostlikely key.

As used in this disclosure, the term “over” is used to describe thepositioning of one or more objects (for example, a finger, thumb, orstylus) over, on, or near a location on a touch screen. In someembodiments, this object need not come into contact with the touchscreen for the object's position to be determined. In other embodiments,the object described as “over” the touch screen may be in contact withthe surface of the touch screen. In some embodiments, the objectdetermined to be “over” a location of the touch screen may not actuallybe positioned directly over the touch screen location, but determined tobe “over” the location on the touch screen, for example, by a positioncorrection module of the text entry device or touch screen.

The disclosed technology includes various approaches to improving typingaccuracy or typing speed when using devices having a touch screen byusing suggestion candidates to augment other input devices. Thesesuggestion candidates are typically represented in a candidates area,which need not be permanently reserved for that purpose, or can appearin varying location on the touch screen. After entering one or morewords to form a text entry, the user can review the text entry byviewing the touch screen and deciding whether to select word(s) for“suggestions.” Although some examples disclosed herein describe “a word”or “a selected word,” it should be understood that in some examples,selecting a word can include but is not limited to selecting a singleword with a single touch screen input, selecting multiple words of aphrase with a single touch screen input, or selecting multiple words ofa phrase using touch screen input comprising plural single touch screeninputs. For example, auto-correction or unexpected-key feedback can begenerated for a single word, or for a phrase comprising multiple wordsand spaces, but that are related in some way.

In some examples, after an indicator is provided, one or more suggestioncandidates are displayed on a touch screen display. The suggestioncandidates can be presented as “buttons” which include a word related tothe word selected by the user. Suggestion candidates can be determinedto be related to the selected word using a candidate generation module,which can use a dictionary, thesaurus, common speller applicationprogramming interface, input history data source, or other sources ormethods to generate suggestion candidates. The candidate generationmodule can also determine the rank order in which suggestion candidatesare presented. For example, the suggestion candidates can be presentedfrom left to right, with the suggestion candidate determined to be themost likely presented furthest to the left, and the least likelysuggestion candidate presented furthest to the right. The user reviewsthe suggestion candidates, and selects one of the candidates forreplacement using a single touch screen input over the desiredsuggestion candidate on the touch screen.

As used in this disclosure, a single touch screen input refers to theinput received when a user positions an object over the surface of atouch screen such that the touch screen device can determine theposition of the object. In some embodiments, the object can be theuser's finger or thumb. In other embodiments, the object can be a stylusor puck. In some embodiments, the single touch screen input is receivedafter the user “taps” the touch screen over a word or suggestioncandidates. In other embodiments, the single touch screen input isreceived when the user presses the screen with a finger, thumb, orstylus. Receiving a single touch screen input is sufficient to determinewhich suggestion candidate the user is indicating on the touch screen—noadditional keyboard input, mouse input, trackball input, voice input, oradditional touches are necessary. Using a single touch screen input todetermine user selections simplifies the input process and allows forthe fast correction of text entries without the need to use submenus,popup menus, or additional input devices.

II. Example Mobile Device

FIG. 1 is a system diagram depicting an exemplary mobile device 100including a variety of optional hardware and software components, showngenerally at 102. Any components 102 in the mobile device cancommunicate with any other component, although not all connections areshown, for ease of illustration. The mobile device can be any of avariety of computing devices (e.g., cell phone, smartphone, handheldcomputer, Personal Digital Assistant (PDA), etc.) and can allow wirelesstwo-way communications with one or more mobile communications networks104, such as a cellular or satellite network.

The illustrated mobile device 100 can include a controller or processor110 (e.g., signal processor, microprocessor, ASIC, or other control andprocessing logic circuitry) for performing such tasks as signal coding,data processing, input/output processing, power control, and/or otherfunctions. An operating system 112 can control the allocation and usageof the components 102 and support for one or more application programs114. The application programs can include common mobile computingapplications (e.g., email applications, calendars, contact managers, webbrowsers, messaging applications), or any other computing application.

The illustrated mobile device 100 can include memory 120. Memory 120 caninclude non-removable memory 122 and/or removable memory 124. Thenon-removable memory 122 can include RAM, ROM, flash memory, a harddisk, or other well-known memory storage technologies. The removablememory 124 can include flash memory or a Subscriber Identity Module(SIM) card, which is well known in GSM communication systems, or otherwell-known memory storage technologies, such as “smart cards.” Thememory 120 can be used for storing data and/or code for running theoperating system 112 and the application programs 114. Example data caninclude web pages, text, images, sound files, video data, or other datasets to be sent to and/or received from one or more network servers orother devices via one or more wired or wireless networks. The memory 120can be used to store a subscriber identifier, such as an InternationalMobile Subscriber Identity (IMSI), and an equipment identifier, such asan International Mobile Equipment Identifier (IMEI). Such identifierscan be transmitted to a network server to identify users and equipment.

The memory 120 can also be used for the candidate sources 116, which areused for generating and suppressing auto-corrections and generationsuggestion candidates. Candidate sources 116 can include but are notlimited to: a system dictionary, a user dictionary, a common spellerapplication programming interface (CSAPI), touch models, and an inputhistory data source.

The mobile device 100 can support one or more input devices 130, such asa touch screen 132, microphone 134, camera 136, physical keyboard 138,and/or trackball 140 and one or more output devices 150, such as aspeaker 152 and a display 154. Other possible output devices can includehaptic output devices such as a piezoelectric transducer 156 or othersuitable device. Some devices can serve more than one input/outputfunction. For example, touch screen 132 and display 154 can be combinedin a single input/output device.

A wireless modem 160 can be coupled to an antenna (not shown) and cansupport two-way communications between the processor 110 and externaldevices, as is well understood in the art. The modem 160 is showngenerically and can include a cellular modem for communicating with themobile communication network 104 and/or other radio-based modems (e.g.,Bluetooth 164 or Wi-Fi 162). The wireless modem 160 is typicallyconfigured for communication with one or more cellular networks, such asa GSM network for data and voice communications within a single cellularnetwork, between cellular networks, or between the mobile device and apublic switched telephone network (PSTN).

The mobile device can further include at least one input/output port180, a power supply 182, a satellite navigation system receiver 184,such as a Global Positioning System (GPS) receiver, an accelerometer186, and/or a physical connector 190, which can be a USB port, IEEE 1394(FireWire) port, and/or RS-232 port. The illustrated components 102 arenot required or all-inclusive, as any components can deleted and othercomponents can be added.

III. Example Computing Environment

FIG. 2 illustrates a generalized example of a suitable computingenvironment 200 in which described embodiments, techniques, andtechnologies may be implemented. For example, the computing environment200 can implement unexpected-key feedback, auto-correction notification,invoking likely next phrases, random playback of click sounds, andwaiting for a keystroke latency before presenting suggestion candidates,as described below.

The computing environment 200 is not intended to suggest any limitationas to scope of use or functionality of the technology, as the technologymay be implemented in diverse general-purpose or special-purposecomputing environments. For example, the disclosed technology may beimplemented with other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. The disclosed technology may also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

With reference to FIG. 2, the computing environment 200 includes atleast one central processing unit 210 and memory 220. In FIG. 2, thismost basic configuration 230 is included within a dashed line. Thecentral processing unit 210 executes computer-executable instructionsand may be a real or a virtual processor. In a multi-processing system,multiple processing units execute computer-executable instructions toincrease processing power and as such, multiple processors can berunning simultaneously. The memory 220 may be volatile memory (e.g.,registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flashmemory, etc.), or some combination of the two. The memory 220 storessoftware 280 and candidate sources 285 that can, for example, implementthe technologies described herein. A computing environment may haveadditional features. For example, the computing environment 200 includesstorage 240, one or more input devices 250, one or more output devices260, one or more communication connections 270, and one or more touchscreens 290. An interconnection mechanism (not shown) such as a bus, acontroller, or a network, interconnects the components of the computingenvironment 200. Typically, operating system software (not shown)provides an operating environment for other software executing in thecomputing environment 200, and coordinates activities of the componentsof the computing environment 200.

The storage 240 may be removable or non-removable, and includes magneticdisks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any othermedium which can be used to store information and that can be accessedwithin the computing environment 200. The storage 240 storesinstructions for the software 280 and candidate sources 285, which canimplement technologies described herein.

The input device(s) 250 may be a touch input device, such as a keyboard,keypad, mouse, pen, or trackball, a voice input device, a scanningdevice, or another device, that provides input to the computingenvironment 200. For audio, the input device(s) 250 may be a sound cardor similar device that accepts audio input in analog or digital form, ora CD-ROM reader that provides audio samples to the computing environment200. The output device(s) 260 may be a display, printer, speaker,CD-writer, or another device that provides output from the computingenvironment 200. The touch screen 290 can act as an input device(receiving touch screen input) and as an output device (displaying thetext entry area, suggestion candidates area, and/or touch keyboard).

The communication connection(s) 270 enable communication over acommunication medium (e.g., a connecting network) to another computingentity. The communication medium conveys information such ascomputer-executable instructions, compressed graphics information, orother data in a modulated data signal.

Computer-readable media are any available media that can be accessedwithin a computing environment 200. By way of example, and notlimitation, with the computing environment 200, computer-readable mediainclude memory 220, storage 240, communication media (not shown), andcombinations of any of the above.

Computer-readable media are any available media that can be accessedwithin a computing environment 200. By way of example, and notlimitation, with the computing environment 200, computer-readable mediainclude memory 220 and/or storage 240. As should be readily understood,the term computer-readable storage media includes the media for datastorage such as memory 220 and storage 240, and not transmission mediasuch as modulated data signals.

IV. Example Text Entry Device

FIG. 3 illustrates a generalized example of a suitable implementationenvironment 300 of a text entry device 305 connected to a computingcloud 325. The text entry device 305 includes several modules stored ona computer-readable storage medium 310, including a text input module330 for receiving text entry input, touch input module 332 for receivingtouch screen input from a touch screen (not shown), and an output module334 for providing output to a touch screen. The communication module 320adapts the text entry device 305 so that it can communicate with serviceproviders located in the cloud 325. The computer-readable storage medium310 also includes an indicator generation module 340 for generatingfeedback semaphores and other indicators, a correction module 342 forchecking and correcting text entries, and a candidate generation module344 for generating suggestion candidates. Indicator generation module340, correction module 342, and candidate generation module 344 cancommunicate with multiple modules to determine correction and suggestioncandidates, including a grammar checking module 350, a system dictionarymodule 351, a user dictionary module 352, a CSAPI (Common Speller API)module 353, and an IHDS (input history data source) module 354. In someembodiments, one or all of these source modules 350-354 can be providedby a service provider in an alternate location 380 in the cloud 325.

FIG. 15 illustrates a generalized example of a suitable implementationenvironment 1500 in which described embodiments, techniques, andtechnologies may be implemented.

In example environment 1500, various types of services (e.g., computingservices) are provided by a computing cloud 1510. For example, the cloud1510 can comprise a collection of computing devices, which may belocated centrally or distributed, that provide cloud-based services tovarious types of users and devices connected via a network such as theInternet. The implementation environment 1500 can be used in differentways to accomplish computing tasks. For example, some tasks (e.g.,processing user input and presenting a user interface) can be performedon local computing devices (e.g., connected devices 1530-1532) whileother tasks (e.g., storage of data to be used in subsequent processing,including candidate sources) can be performed in the cloud 1510.

In example environment 1500, the cloud 1510 provides services forconnected devices 1530-1532 with a variety of screen capabilities.Connected device 1530 represents a device with a computer screen (e.g.,a mid-size screen). For example, connected device 1530 could be apersonal computer such as desktop computer, laptop, notebook, netbook,or the like. Connected device 1531 represents a device with a mobiledevice screen (e.g., a small size screen). For example, connected device1531 could be a mobile phone, smart phone, personal digital assistant,tablet computer, and the like. Connected device 1532 represents a devicewith a large screen. For example, connected device 1532 could be atelevision screen (e.g., a smart television) or another device connectedto a television (e.g., a set-top box or gaming console) or the like. Oneor more of the connected devices 1530-1532 can include touch screencapabilities. Devices without screen capabilities also can be used inexample environment 1500. For example, the cloud 1510 can provideservices for one or more computers (e.g., server computers) withoutdisplays.

Services can be provided by the cloud 1510 through service providers1520, or through other providers of online services (not depicted). Forexample, cloud services can be customized to the screen size, displaycapability, and/or touch screen capability of a particular connecteddevice (e.g., connected devices 1530-1532).

In example environment 1500, the cloud 1510 provides the technologiesand solutions described herein to the various connected devices1530-1532 using, at least in part, the service providers 1520. Forexample, the service providers 1520 can provide a centralized solutionfor various cloud-based services. The service providers 1520 can manageservice subscriptions for users and/or devices (e.g., for the connecteddevices 1530-1532 and/or their respective users).

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed methods can be used in conjunction with other methods.

Any of the disclosed methods can be implemented as computer-executableinstructions stored on one or more computer-readable media (e.g.,non-transitory computer-readable media, such as one or more opticalmedia discs, volatile memory components (such as DRAM or SRAM), ornonvolatile memory components (such as hard drives)) and executed on acomputer (e.g., any commercially available computer, including smartphones or other mobile devices that include computing hardware). Any ofthe computer-executable instructions for implementing the disclosedtechniques as well as any data created and used during implementation ofthe disclosed embodiments can be stored on one or more computer-readablemedia (e.g., non-transitory computer-readable media). Thecomputer-executable instructions can be part of, for example, adedicated software application or a software application that isaccessed or downloaded via a web browser or other software application(such as a remote computing application). Such software can be executed,for example, on a single local computer (e.g., any suitable commerciallyavailable computer) or in a network environment (e.g., via the Internet,a wide-area network, a local-area network, a client-server network (suchas a cloud computing network), or other such network) using one or morenetwork computers.

For clarity, only certain selected aspects of the software-basedimplementations are described. Other details that are well known in theart are omitted. For example, it should be understood that the disclosedtechnology is not limited to any specific computer language or program.For instance, the disclosed technology can be implemented by softwarewritten in C++, Java, Perl, JavaScript, Adobe Flash, or any othersuitable programming language. Likewise, the disclosed technology is notlimited to any particular computer or type of hardware. Certain detailsof suitable computers and hardware are well known and need not be setforth in detail in this disclosure.

Furthermore, any of the software-based embodiments (comprising, forexample, computer-executable instructions for causing a computer toperform any of the disclosed methods) can be uploaded, downloaded, orremotely accessed through a suitable communication means. Such suitablecommunication means include, for example, the Internet, the World WideWeb, an intranet, software applications, cable (including fiber opticcable), magnetic communications, electromagnetic communications(including RF, microwave, and infrared communications), electroniccommunications, or other such communication means.

The disclosed methods, apparatus, and systems should not be construed aslimiting in any way. Instead, the present disclosure is directed towardall novel and nonobvious features and aspects of the various disclosedembodiments, alone and in various combinations and subcombinations withone another. The disclosed methods, apparatus, and systems are notlimited to any specific aspect or feature or combination thereof, nor dothe disclosed embodiments require that any one or more specificadvantages be present or problems be solved.

V. Example Touch Screen Text Entry Device

FIG. 4 depicts an exemplary embodiment 400 of a text entry device 401having a touch screen 402. The touch screen 402 includes a display areafor a touch screen keyboard 420, a suggestion candidates area 450, and atext display area 403. The text display area 403 is shown displaying aphrase “smsing while dr” 404. In some embodiments, only a portion of thetext input is displayed because of, for example, screen sizelimitations. A carat (cursor) 406 is also shown in the text entry area403. In some embodiments, the carat 406 can be placed at variouspositions in the text entry area using a single touch screen input. Insome embodiments, the touch screen 402 has a substantially planarsurface, and the display capability can be implemented using LED, LCD,electronic ink, DLP, Plasma, CRT, or other suitable display technology.

The text entry device 401 has a touch screen 402 that displays a touchscreen keyboard 420 having several keys 424, 426, 428, 430, 432, 434,etc. Some of the keys, including the backspace key 430, return key 432,and space key 434 are also designated as delimiter keys. As shown, thetouch screen keyboard displays the keys 424, 426, 428, 430, 432, 434,etc. as images on the touch screen 402, as the user's finger 442 istouching the key “s” 428. The touch screen can include capacitive,resistive, inductive, or other suitable technologies for determining theposition of one or more touch inputs detected over the surface of thekeyboard and converting this touch input into text input. In someembodiments, the touch input is created using a stylus or puck, while inother embodiments the touch input can be created using a finger orthumb. In other embodiments, the touch screen keyboard 420 can beimplemented as a hardware keyboard including mechanical keys.

The suggestion candidates area 450 is depicted in FIG. 4 showing threesuggestion candidates, including “drive” 455, “drug,” “draft,” “drop,”and “driving.” As shown, the suggestion candidates area 450 includeswidgets directly above the keyboard area containing word prediction anddisambiguation candidates. Placing the suggestion candidates area 450close to the touch screen keyboard 420 can improve typing speed andreduce errors by allowing users to maintain their focus of attentionnear the keyboard area while correcting a phrase 410 in the text entryarea 403. In other embodiments, the suggestion candidates area 450 canbe placed nearby the phrase 404 in the text entry area 403.

The location of the text entry area 403, the keyboard 420, etc., can bevaried based on the particular implementation and design.

FIGS. 5A-5C illustrate additional exemplary implementations 500, 530,560 of the disclosed technology. FIG. 5A depicts an exemplaryimplementation 500 of a text entry device, which includes a touch screen506 displaying a window 510, including a designated text entry areadisplaying the phrase “smsing while driving” 512. The touch screen 506is also displaying a candidates area 520, and a touch screen keyboard524. A finger 522 is shown selecting the candidate “driving” in thecandidates area 520.

FIG. 5B depicts an exemplary implementation 530, which includes a mobiledevice 531 having a touch screen 532 with a text entry area 534(displaying the phrase “smsing while driving” 536), a suggestioncandidates area 542, and a hardware keyboard 540.

FIG. 5C depicts an exemplary implementation 570 of a text entry device,which includes a personal computer (PC) device having a display screen571 and a keyboard 580. The display screen 571 is shown displaying thephrase “smsing while dr” 575 in a window 574. Also shown is a suggestioncandidates area 586 displaying candidates “drive,” “draft,” “drop,” and“driving.” In some examples, the text entry device is not limited toreceiving text input using a touch screen keyboard, but can also usehardware keyboards, handwriting recognition, or speech recognition toreceive text input.

Methods and apparatus for performing handwriting recognition can includebut are not limited to those based on: Bayesian networks, neural nets,hidden Markov models, or k-nearest-neighbor approaches. Methods andapparatus for performing speech recognition can include but are notlimited to those based on a dynamic time warping approach or hiddenMarkov models.

VI. Example Indicators, Signals, and Feedback Semaphores

Feedback and guidance signals are examples of indicators that can beimplemented in many ways. For example, mobile tactile feedback, such aspiezoelectric tactile displays that responds to a stylus, or a stylusthat produces tactile feedback, can be employed. For touch screenkeyboards in particular, artificial tactile feedback via vibrationactuators can improve text entry performance in both static environmentsand mobile environments (for example, typing in a moving vehicle such asan automobile or train). In some examples, keyboard events can bedefined including a fingertip-over event to signal when a fingertip hastouched a key, a fingertip-click event to signal that a key has beenregistered, a fingertip-slip event to signal when the fingertip movedover the edge of a key, and a home-key event for indicating where thehome keys are located on the touch screen keyboard. For example, in anEnglish-language keyboard, the home keys of “F” and “J,” one for eachfinger, can be associated with haptic feedback that is provided toindicate a user's fingers are over the home keys. Using these events canimprove tactile touch screen keyboard accuracies obtained to close tothose obtained using a physical keyboard. Examples of using hapticfeedback with a touch screen keyboard may be found in: Brewster, et al.,“Tactile feedback for mobile interactions,” Proc. of SIGCHI Conf. onHuman Factors in Computing Systems, pp. 159-162. (2007); Brown &Brewster, “Multidimensional Tactons for Non-visual Information Displayin Mobile Devices,” Proc. of MobileHCI Conf. 2004, pp. 231-238 (2004);and Hoggan et al., “Investigating the Effectiveness of Tactile Feedbackfor Mobile Touchscreens,” Proc. of the 26th Int'l Conf. on Human Factorsin Computing Systems, pp. 1573-1582 (2008).

Although employing different keyboard layouts can assist text entry ontouch screen keyboards, these layouts have not yet become widespread. Itis desirable to create semaphores for a touch screen keyboard that canbe marketed widely.

In addition or instead of semaphores created using tactile feedback,similar results can be achieved by providing visual and auditory cueswith a standard touch screen keyboard. While some users complain thattheir fingertips obfuscate visual feedback provided using a softkeyboard, visual and auditory feedback for the three keyboard eventsdiscussed above can be employed. Some examples of visual/audio feedbackincludes: a tooltip balloon that appears and disappears at thefingertip-over and fingertip-slip events, respectively, and an audibleclick is played at fingertip-click events.

Obstacles to providing effective visual and auditory cues in mobilesettings include small screen size, outside noise, social restrictionsand other circumstantial demands. Moreover, it has not been empiricallyestablished that artificial tactile feedback alone can impart enoughinformation to realize full parity with physical keyboards. Furthermore,some users find tactile feedback to be annoying. Because individualdifferences may account for users' preference of one modality overanother, touch screen keyboards employing a variety of tactile, visualand auditory semaphores, and combinations thereof, can be used.

Certain examples of the disclosed technology include touch screenkeyboards using “typing semaphores.” As described further below, typingsemaphores can provide multimodal feedback and/or guidance signals thathelp users discover and avoid typing errors. Multimodal feedback refersto the use of visual, auditory, and/or haptic (touch output)communication that are received be a text entry device user to indicateinformation such as potential errors, the occurrence of events such asauto-correction, or the availability of suggestion candidates.

A conventional semaphore is a system for conveying information by meansof hand-held flags or lights. Widely adopted by the maritime world inthe 1800s, semaphores are still employed today to safely guide airplanesonto naval vessels. Typing semaphores can be viewed as multimodalsignals that guide users to enter text correctly—by analogy, a kind offlight control for typing. State of the art touch screen keyboards makeit too easy for users to “crash,” expecting them to edit their textthereafter, which can be mentally disruptive, time-consuming, andtaxing.

The purpose of typing semaphores is to improve text entry userexperience and performance as users type (e.g., in real-time) orotherwise provide input data using both feedback signals to alert userto possible typing errors when they occur, and guidance signals toprevent future errors.

VII. Example Application Using Indicators and Suggestion Candidates

FIG. 6 is a flow chart 600 that outlines a method of generating possiblecorrections for input data, providing indicators, including feedbacksemaphores that indicate the availability of the possible corrections,and displaying suggestion candidates. After displaying the suggestioncandidates, an input is received selecting one of the suggestioncandidates, and the input data is modified with one or more wordsassociated with the selected suggestion candidate.

At process block 610, input data comprising one or more input word(s) isreceived from a source such as a touch screen keyboard. In someembodiments, the first input data includes text characters, text words,position data for key presses on a touch screen keyboard, typing speeddata, correction data, and/or touch screen orientation data.

At process block 620, one or more feedback semaphores are provided toindicate that one or more suggestion candidates are available for one ormore of the input words. The feedback semaphores can include variousindicators or signals to alert a user while maintaining focus in thekeyboard area by, for example, highlighting one or more keys of akeyboard, playing back a sound using a speaker, generating hapticfeedback (e.g., vibrating a handheld device), or displaying a tooltipballoon.

At process block 630, one or more suggestion candidates are provided toa user (e.g., using a touch screen display). Each of the suggestioncandidates is associated with an alternative word for one or more of theinput words. In some examples, the suggestion candidates are associatedwith alternative words for only one of the input words, while in otherexamples, suggestion candidates are associated with alternative wordsfor more than one of the input words. An exemplary display of asuggestion candidate includes displaying a button using a touch screendisplay, where the button includes the text of an associated alternativeword within the boundary of the button.

At process block 640, an input selecting one of the suggestioncandidates is received. For example, the selection can received using asingle touch screen input created by a user pressing a finger over abutton associated with the desired touch screen candidate on a touchscreen display.

At process block 650, the input data is modified using the alternativeword associated with the selected suggestion candidate. In someexamples, the alternative word is used to replace the input word in theinput data. In other examples, the alternative word is added to theinput data preceding or subsequent to the input word. In some examples,the “alternative” word is a word that was previously auto-corrected, andthe alternative word is therefore used to effectively undo a wordinserted in the input data using an auto-correction routine. Thus, asdescribed above, a quick and effective way of providing suggestioncandidates and indicators of their availability is provided that allowsusers to quickly modify input data using the suggestion candidateswithout moving their focus of attention away from a touch screenkeyboard.

VIII. Example Unexpected Key Feedback

Because touch screen keyboards lack tactile feedback, it is difficultfor users to tell when they hit their desired keys. Consequently, theyhave to pay more attention to the keyboard area than when using hardwarekeyboards. FIGS. 7A-7E depict a method 700 for providing visual, audio,and tactile feedback to a user that has pressed an expected key on amobile device 702. FIG. 8 is a flow chart 800 corresponding to themethod 700 depicted in FIGS. 7A-7E.

As shown in FIG. 7A, a mobile device 702 comprises a text display area704, suggestion candidates area 708, and touch screen keyboard 710. Thetext display area 704 is shown displaying a phrase “smsing whil” 706,with a carat (cursor) 712 immediately after the phrase as the userenters it on the keyboard 710.

FIG. 7B depicts the mobile device 702 providing the user with unexpectedkey feedback semaphores as an unexpected key, “r” 726, is pressed with auser's finger 722. The types of feedback semaphores shown include: atooltip balloon 728 displaying the unexpected key, an audio “clunkclunk” sound 740 (played back over a speaker coupled to the mobiledevice), haptic feedback (e.g., a vibration created by an oscillatingtransducer coupled to the mobile device) indicated by squiggle lines 755and 756, suggestion candidates “whip” 730, “whilst” 731, and “while”732, and a highlighted key indicator shown as shading of the unexpected“r” key 726. The unexpected “r” key 726 is designated as unexpectedbased on the current context of the text input data shown on the textdisplay area 704 (e.g., the word “whilr” does not exist in the Englishlanguage). Here, the suggestion candidates 730-732 are actually based onthe data input phrase “smsing whil” 706 before the “r” was received, butas shown in FIG. 7C, will be updated based on the unexpected key input.

FIG. 7C depicts the mobile device 702 shortly after providing one ormore feedback semaphores as discussed above. As shown, the phrase 742has been updated with the unexpected key “r,” and the carat 712 isplaced immediately after the phrase. Further, suggestion candidates“while” 750, “white” 751, and “wholesale” 752 have been provided basedon the phrase 742. In addition, designated delimiter keys including aspace bar 758 and an enter key 759 have been highlighted to indicatethat the highlighted suggestion candidate will replace the word “whilr”if one of the delimiter keys is pressed next. Alternatively, the usercan press one of the suggestion candidates 750-752 in order to selectthe corresponding word shown on the candidate button to replace the word“whilr.”

FIG. 7D depicts the mobile device 702 as a user uses a finger 723 toprovide a single touch screen input over a delimiter key (the space bar758).

FIG. 7E depicts the mobile device 702 after receiving the delimiter key.As shown, the phrase 782 has been updated to replace the word “whilr”with the word “while,” which was the word associated with thehighlighted suggestion candidate 750. The carat 712 has also beenadvanced one space past the phrase 782.

The manner of providing feedback semaphores used can be selected basedon their effectiveness in alerting users without annoying them. Forexample, when tooltip balloons (e.g., tooltip balloon 728) appear abovethe keys of the keyboard 710, users do not always see these visual cues,especially if they are typing quickly. It is desirable to use feedbacksemaphores that can alert users to unexpected key presses so that theycan keep their focus on the keyboard 710, or immediately switch theirfocus of attention to the candidates area 708. A key is designated“unexpected” when the letters entered so far (e.g., up to the previousword boundary) do not match the prefix of any word in a candidate source(e.g., a dictionary, CSAPI, or input history data source).

FIGS. 7F-7H depict different designs 760, 770, and 780 of highlighting atouch screen that can be used to provide unexpected-key feedbacksemaphores to a user. In FIGS. 7F-7H, a mobile device 702 is shown asthe user types the “r” key after entering the phrase “smsing whil” 762.As shown in the design 760 of FIG. 7F, a shaded tooltip balloon 766 andboth the background and keys of the touch screen keyboard 764 arehighlighted to provide an unexpected key feedback semaphore. Similarly,the design 770 of FIG. 7G depicts the mobile device 702, but with ashaded tooltip balloon 776 and only the background of the touch screenkeyboard 774 highlighted. Finally, the design 780 of FIG. 7H depicts themobile device 702, but with a shaded tooltip balloon 786 and only theborder of the background of the touch screen keyboard 784 highlighted.The highlighting and shading shown in FIGS. 7A-7H can be based on usinga pattern and/or color(s), and can slowly fade back to the original keycolors over time to present a more pleasing visual appearance.

Usability studies have indicated that some users found the design 760shown in FIG. 7F, including a shaded keyboard 764, shaded “r” key, andshaded tooltip balloon 766, “too disruptive.” Although the design 770 ofFIG. 7G, including shading for only the background of the keyboard 774,shaded “r” key, and shaded tooltip balloon 776, was found to be lessdisruptive, users still found this design more distracting than thedesign shown in FIG. 7H, which includes only a shaded border around thekeyboard 784, shaded “r” key, and shaded tooltip balloon 786. Thus, byperforming usability studies, desirable combinations of visual andauditory feedback that are agreeable to users can be determined.

FIG. 8 is a flow chart 800 further detailing the method 700 shown inFIGS. 7A-7E. FIG. 8 depicts process blocks for receiving input data,evaluating the data using one or more candidate sources to determineunexpected input, and based on the evaluating, displaying feedbacksemaphores and suggestion candidates allowing a word to be replaced witha selected suggestion candidate.

At process block 810, input data is received from a source such as atouch screen keyboard. In some embodiments, the first input dataincludes text characters, text words, position data for key presses on atouch screen keyboard, typing speed data, correction data, and/or touchscreen orientation data.

At process block 820, the input data is analyzed and compared againstone or more of the following candidate sources: a system dictionary; auser dictionary; a common speller application programming interface(CSAPI); or an input history data source that can include previouslygenerated input history data for one or more users including wordprobability data, key probability data, edit distance data, and touchmodel probability data. In some examples, the analysis and comparison isrepeated continuously for each character of text input, while in otherexamples, the analysis and comparison may only occur after an event.Examples of events include receiving a delimiter key (e.g., a space,backspace, enter, ESC, or other designated key(s)) or determining alatency in user typing speed that exceeds a threshold value.

At process block 830, the analysis and comparison data from processblock 820 is evaluated to determine if any unexpected input data wasreceived. If no unexpected input data is received (e.g., all the wordsof the text input data can be found in one or more of the candidatesources used at process block 820), the method proceeds to process block810 in order to continue receiving input data. If unexpected input datais detected, the method proceeds to process block 840. In someembodiments, a word can be designated as unexpected input data eventhough the word may exist in one of the candidate sources, for example,using a grammar checker module. An unusual word such as “whilst” maystill be designated as unexpected input data because other similarwords, for example, “while” have a higher probability of correctness,based on word probability data, key probability data, edit distancedata, and touch model probability data. In some examples, the feedbacksemaphores for unusual words designated as unexpected may be modified incomparison to feedback semaphores used for words that do not exist inany of the candidate sources at all.

At process block 840, an indicator such as a feedback semaphore isprovided in order to alert a user that suggestion candidates areavailable. As discussed above, the feedback semaphores provided caninclude visual displays, including the highlighting of touch screenkeyboard keys or keyboard background; audio playback, such as adistinctive “clunk clunk” sound (which indicates to the user thatsomething may be incorrect in the input data), or haptic feedback.

At process block 850, suggestion candidates are provided, using forexample, a candidates area on a touch screen display. Each suggestioncandidate is associated with an alternative word that will be used tomodify the input data if that particular suggestion candidate isselected. In some examples, one of the candidates can by displayeddifferently (or highlighted differently) than other candidates toindicate that it is the most likely suggestion candidate. Thisdistinguishing display not only provides a visual cue to the user, butcan also be used to indicate that the input data will be modified withthe suggestion candidate if other designated keys are pressed, forexample, a delimiter key such as the space key, punctuation keys, or thereturn or enter keys.

At process block 860, the selection (or absence of a selection) of oneof the suggestion candidates is detected. Suggestion candidates can beselected several ways, including receiving a single touch screen inputover the suggestion candidate on the touch screen display, or byreceiving a keystroke for a designated delimiter key. If no suggestioncandidate is selected, the method proceeds to process block 810 in orderto receive more input data. If a suggestion candidate selection isdetected, the method proceeds to process block 870.

At process block 870, the input data is modified using an alternativeword associated with the selected suggestion candidate. In someexamples, the alternative word is used to replace the input word in theinput data. In other examples, the alternative word is added to theinput data preceding or subsequent to the input word. In some examples,the “alternative” word is a word that was previously auto-corrected, andthe alternative word is therefore used to effectively undo a wordinserted in the input data using an auto-correction routine. Thus, asdescribed above, a quick and effective way of providing suggestioncandidates and indicators of their availability is provided that allowsusers to quickly modify input data using the suggestion candidateswithout moving their focus of attention away from a touch screenkeyboard.

IX. Example Auto-Correction Feedback

Users often keep their focus of attention on the keyboard area whentyping on a touch screen keyboard, which often causes users to fail tosee auto-corrections that may be replacing legitimate words, such asnames and technical terms that do not exist in the dictionary. This canlead to tremendous frustration, especially if users do not notice thetext replacements until much later.

FIGS. 9A-9D depict a method 900 of providing the user with anauto-correction feedback semaphore. FIG. 10 is a flowchart 1000corresponding to the method 900 depicted in FIGS. 9A-9D.

As shown in FIG. 9A, a mobile device 902 comprises a text display area904, suggestion candidates area 910, and touch screen keyboard 908. Thedisplay area 904 is shown displaying a phrase “smsing” 920 after a userhas typed the unknown word “smsing,” but before typing a delimiter, so acarat 916 is displayed immediately after the phrase. Shown in thesuggestion candidates area 910 are the suggestion candidates “ending”912, “facing” 913, and “smashing” 914.

FIG. 9B depicts the mobile device 902 as the user presses the space bar932 with a finger 934, which causes the space bar to immediately behighlighted red, the word “smsing” in the phrase 920 to be replaced witha replacement word “ending,” 922 a “swish” sound 950 to be played over aspeaker, and the mobile device 902 to vibrate, as shown by squigglelines 955 and 956. The carat 916 is advanced by a space, and the mobiledevice also displays the suggestion candidates “smsing” 940, “facing”941, and “smashing” 942 in the candidates area. The candidate “smsing”940 is the word that was just replaced by auto-correction, and thecandidate 940 is colored green to indicate that it is the replaced word.

FIG. 9C depicts the mobile device 902 after the user has decided to undothe auto-correction. To do so, the user uses a finger 962 to make asingle touch screen input over the suggestion candidate “smsing” 966,which will undo the auto-correction.

FIG. 9D depicts the mobile device 902 after the auto-correction has beenundone. As shown, the word “ending” has been replaced with the originalword “smsing” 982, the suggestion candidates area 910 is cleared, andthe carat 916 remains a single space past the replaced word.

FIG. 10 is a flow chart 1000 further detailing the method 900 shown inFIGS. 9A-9D. At process block 1010, a mobile device receives input datafrom a source such as a touch screen keyboard. In some embodiments, thefirst input data includes text characters, text words, position data forkey presses on a touch screen keyboard, typing speed data, correctiondata, and/or touch screen orientation data.

At process block 1020, the input data is analyzed to determine if adesignated delimiter key (e.g., a space key or an enter key) has beenreceived in the input data. If not, then the method proceeds processblock 1010 to receive more input data. If a delimiter key was receivedin the input data, the method proceeds to process block 1030.

At process block 1030, the last word of the input data is comparedagainst one or more candidate sources, and an auto-correction is appliedto replace a word designated as incorrect with a replacement word from acandidate source.

At process block 1040, one or more auto-correction feedback semaphoresare displayed or played back, in a manner that is likely to keep theuser's focus of attention on the keyboard. For example, a “swish” soundcan be played, haptic feedback provided, and red highlighting displayedon the delimiter key that was pressed by the user. The feedbacksemaphores are displayed in a manner that does not require the user toshift their focus of attention away from the keyboard, and therefore theuser is able to more easily recognize that an auto-correction hasoccurred, as well as to easily view suggestion candidates that will bepresented at process block 1050.

At process block 1050, one or more suggestion candidates are presentedto the user, including the original word that was auto-corrected. One ofthe suggestion candidates is also highlighted using, for example,shading or a different color. The highlighted suggestion candidate isdesignated as the most likely substitution that the user will select. Insome examples, the highlighted suggestion candidate is original wordthat was auto-corrected. In other examples, the highlighted suggestioncandidate is a word from a candidate source designated to be the mostlikely substitution. The method proceeds to process block 1060 afterreceiving touch screen input.

At process block 1060, the method determines whether the user selected adelimiter key. If a delimiter key was pressed, the method proceeds toprocess block 1070, where the word inserted by auto-correction atprocess block 1030 is itself replaced with the word shown as thehighlighted suggestion candidate, and the method then proceeds toprocess block 1010, where more input data is received. If a delimiterkey was not pressed, the method proceeds to process block 1080.

At process block 1080, the method determines whether the user selected asuggestion candidate (e.g., by providing a single touch screen inputover the suggestion candidate). If a suggestion candidate is selected,the method proceeds to process block 1090, where the word inserted byauto-correction at process block 1030 is itself replaced with the wordassociated with the selected suggestion candidate. If a suggestioncandidate is not pressed, the method proceeds to process block 1010 toprocess the selected key (e.g., by adding the selected key to the inputdata) and continue receiving input data.

X. Example Invoking Next Phrases

FIGS. 11A-11F depict exemplary methods 1100 and 1150 of providing a userwith predicted next phrases after receiving a completed word of textinput using suggestion candidates. In some examples, users can selecttwo or more suggestion candidates sequentially, thereby improving typingspeed performance. FIG. 12 is a flow chart 1200 illustrating anexemplary implementation of a method corresponding to the methods 1100and 1150 shown in FIGS. 11A-11F.

As shown in FIG. 11A, a mobile device 1102 has a text display area 1104,a suggestion candidates area 1110, and a keyboard 1108. The mobiledevice 1102 is shown after a user has typed one word of the phrase“happy” 1106, and the carat 1116 is positioned immediately after theword “happy.”

FIG. 11B depicts the mobile device 1102 after the user has typed adelimiter key (e.g., the space bar). As shown, three suggestioncandidates “birthday” 1120, “new” 1121, and “go lucky” 1122 aredisplayed in the candidates area 1110. The user is shown using a finger1130 to select the first suggestion candidate 1120.

FIG. 11C depicts the mobile device 1102 after adding an additional nextword “birthday” 1138 after the word “happy” in the phrase 1106 based onthe selected suggestion candidate and positioned the carat 1116 onespace away from the end of the word 1138.

FIG. 11D depicts an exemplary method 1150 including an alternativeselection of a suggestion candidate by a user compared to that shown inFIGS. 11A-11C. FIG. 11D shows a mobile device 1152 entering a delimitercharacter immediately after typing the word “happy” 1154. The carat 1156is positioned one space past the word. A user is shown selecting thesuggestion candidate “new” 1162 with a finger 1165 in the suggestioncandidates area 1160.

FIG. 11E depicts the mobile device 1152 after the user has selected thesuggestion candidates “new” 1162. The word “new” 1170 is added to thetext input after the word “happy,” another search of one or morecandidate sources is performed for the phrase “happy new,” and severalsuggested next words for the phrase “happy new” are presented assuggestion candidates: “year” 1182, “baby” 1183, and “day” 1184. Theuser is shown selecting the suggestion candidate “year” 1182 with afinger 1175.

FIG. 11F depicts the mobile device 1152 after adding the word “year”1190 associated with the selected suggestion candidate 1182 to the textinput, to form the phrase “happy new year.” Another search of the one ormore candidates sources is performed, but no matches are found, so thesuggestion candidates area 1160 is cleared, and the carat 1156 is placedone space from the word “year” 1190.

FIG. 12 is a flow chart 1200 further detailing the exemplary methods1100 and 1150 shown in FIGS. 11A-11F. At process block 1210, input datais received, and the method proceeds to process block 1220, where thepresence or absence of a delimiter character in the input data detected.If no new delimiter key is received in the input data, the methodproceeds to process block 1210 to receive additional input data. If adelimiter key is received, then the method proceeds to process block1230 to generate suggestion candidates for the input data. In someexamples, only the last word typed is used to search for suggestioncandidates, while in other examples, additional words in the input datacan be included in the search.

At process block 1240, the method determines if any suggestioncandidates were generated. If no suggestion candidates are generated,the method proceeds to process block 1210 to receive additional inputdata. If one or more candidates were generated at process block 1230,the method proceeds to process block 1250. At process block 1250, thesuggestion candidates are presented to the user using a touch screendisplay coupled to the mobile device.

At process block 1260, a suggestion candidate selection (or absence of aselection) is detected. Suggestion candidates can be selected severalways, including receiving a single touch screen input over thesuggestion candidate on the touch screen display, or by receiving akeystroke for a designated delimiter key. If no suggestion candidate isselected, the method proceeds to process block 1210 in order to receivemore input data. If a suggestion candidate selection is detected, themethod proceeds to process block 1270.

At process block 1270, the input data is modified using an alternativeword associated with the selected suggestion candidate. In someexamples, the alternative word is used to replace the input word in theinput data. In other examples, the alternative word is added to theinput data preceding or subsequent to the input word. In some examples,the “alternative” word is a word that was previously auto-corrected, andthe alternative word is therefore used to effectively undo a wordinserted in the input data using an auto-correction routine. Aftermodifying the input data, the method immediately proceeds to processblock 1230, where more suggestion candidates are generated based on themodified input data. In this manner, a user can select multiplesuggestion candidates from a changing set of suggestion candidates,thereby improving typing speed and accuracy, especially for highlyfrequent combinations of words and phrases.

XI. Example Use of Typing Speed and Simulating Click Sounds

FIG. 13 is a chart 1300 representing some events that occur during amethod that includes the use of random click sounds played back as theuser presses keys on a touch screen keyboard, as well as the use ofkeystroke latency to determine when to present suggestion candidate to atouch screen keyboard user. FIG. 14 is a flow chart 1400 thatcorresponds to the methods shown in FIG. 13.

Shown in FIG. 13 is a time axis 1330 that indicates when events occur orare received by a mobile device. The user enters several successive keypresses on the touch screen: “b” 1340, “a” 1341, “n” 1342, and “a” 1343in relatively rapid succession. At the same time, the mobile devicerandomly selects and plays a “click” sound that simulates the sound thata key press makes when typing on a hardware keyboard. For example, themobile device can randomly select one of the five different click soundslabeled #1-#5. As shown, the click sounds 1310-1314, etc., are notalways the same for a given key that is pressed. For example, the firstkeypress “a” 1341 is followed by playback of click sound #2 1311, whilethe second keypress “a” 1344 is followed by playback of click sound #31314. Similarly, the two keypresses shown for “n” playback sound #3 1312the first keypress 1342 and sound #5 1313 for the second keypress 1344.By providing playback of random click sounds that sound like a hardwarekeyboard, the user receives more naturalist audio feedback that moreclosely simulates typing on a hardware keyboard.

Also shown in FIG. 13 is the presentation of suggestion candidates basedon the user's keystroke latency. The mobile device monitors a user'skeystroke latency as words are typed, and if the user pauses for alength of time that exceeds a pre-determined threshold, suggestioncandidates are presented for the input data of the most recent wordinput. The techniques for presenting the suggestion candidates aresimilar to those discussed above. In some examples, suggestioncandidates are only presented when the partial data input is determinedto be incomplete, i.e., when the partial input data does not appear in acandidate source such as a dictionary or input history data. In otherexamples, suggestion candidates are presented after a pre-determinedlatency if no delimiter has been received for the word. In someexamples, the user can define the latency threshold value, while othersexamples use a pre-defined threshold value.

As shown, the user types the keys “b” 1340, “a” 1341, “n” 1342, and “a”1343 in relatively rapid succession, followed by a relatively longerpause. After the latency threshold has been reached based on the delayafter the keypress 1343, the mobile device presents several suggestioncandidates for the partial input data “bana,” includes candidates“banana” 1350, “bananas” 1351, and “banal” 1352. As shown, the userdecides to ignore the suggestion candidates and continues typing“na<space>s” followed by another pause after keypress “s” 1345. Anothertime period exceeds the latency threshold value after the keypress “s”1345, and different suggestion candidates 1360-1363 are generated. Thus,the suggestion candidate generation module can consider not only thepartial input word “s,” but also the previous word input “banana.”(e.g., “banana s” is the input being considered by the candidategeneration module). Thus, very accurate suggestion candidates can beprovided even though the user has only entered a single letter “s” ofthe latest input word. As shown, the user selects the candidate “slug”1362 by pressing the “slug” suggestion candidate 1370 on the touchscreen display, and the input data is modified to “banana slug.”

FIG. 14. is a flow chart 1400 further detailing the method shown in FIG.13. At process block 1410, input data is received from a source such asa touch screen keyboard. In some embodiments, the first input dataincludes text characters, text words, position data for key presses on atouch screen keyboard, typing speed data, correction data, and/or touchscreen orientation data.

At process block 1420, a random click sound is selected from acollection of two or more sounds and played back to the user using,e.g., a speaker. As discussed above regarding FIG. 13, the click soundsare not associated with any specific key, and any particular key presson the same key can have multiple random sounds played back.

At process block 1430, the latency between the most recently receivedkeystroke, and last keystroke received immediately preceding thatkeystroke, is measured. At process block 1440, the measured latency iscompared to a pre-determined threshold value. In some examples, thislatency threshold can be selected by the user. In other examples, thethreshold can come from a default value. In other examples, thethreshold can be selected based on a user's previous or average typingspeed. If the measured latency does not exceed the threshold, the methodproceeds to process block 1410, and more input data is received. If themeasured latency does exceed the threshold, the method proceeds toprocess block 1450.

In some examples, the keystroke latency or typing speed data is alsoused to select which types of feedback semaphore(s) are presented. Forexample, if a user is typing relatively rapidly, only one or two moresubtle semaphores are provided (e.g., only delimiter key highlightingcan be used for fast typists). If a user is typing relatively slowly,more semaphores, or semaphores that use stronger visual, audio, and/orhaptic cues are provided (e.g., louder audio, use of haptic force,and/or brighter or different-colored highlighting can be used for slowertypists). Alternatively, or in addition, typing speed data, touch pointdata, device orientation, etc., can also be used to detect whether auser is typing using one or two hands. Thus, different types of feedbacksemaphores can be provided based on whether the user is typing using oneor two hands.

At process block 1450, the input data is analyzed and suggestioncandidates are generated and presented for the input data in a similarfashion to those techniques for generating suggestion candidatesdiscussed above. In some examples, candidates are only generated if theinput data is partial and cannot be found in a candidate generationsource. In other examples, candidates are generated unless the last keypressed is a delimiter key. In some examples, only partial input data isconsidered, while other examples can analyze other words in the inputdata (e.g., the word immediately preceding or the word immediatelysubsequent to the current input word). After the candidates aregenerated, they are presented to the user using, e.g., a candidates areaon a touch screen device.

At process block 1460, the selection (or absence of selection) of one ofthe suggestion candidates is detected. If a suggestion candidateselection is detected, the method proceeds to process block 1470, wherethe partial input data is modified by replacing the input word with theword associated with the selected suggestion candidate, and the methodproceeds to process block 1410 to receive more input data. If asuggestion candidate is not detected as selected, then the methodproceeds to process block 1410 to receive more data.

XII. First Experimental Results

A. Introduction

Experiments were conducted in order to measure the text entryperformance improvement in a controlled experiment using severalsemaphore designs on naïve participants in a usability study.

One of the semaphores studied is called mobile key-trail feedback. Tohelp users keep track of the keys they just typed in the keyboard area,the on-screen keyboard utilizes a feedback technique where pressed keysbriefly light up and gradually fade away.

Another semaphore studied is a guidance semaphore called key-predictionguidance. An example of key-prediction guidance is highlighting orbolding the next likely letter on a keyboard.

The other two semaphores studied were the feedback semaphoresunexpected-key feedback and auto-correction feedback.

B. Experiment Methodology

Four female and seven male participants from a metropolitan area wererecruited by a professional contracting service. Three owned touchscreen phones, five had owned a QWERTY phone at some point in theirlives, and three owned only 12-key numeric phones. The participants camefrom various occupational backgrounds from housewife to IT professional,and were within an age range of 19-39. All participants were compensatedfor their time.

Participants were shown a short phrase on a desktop computer screen fromthe well-known phrase set described in MacKenzie & Soukoreff, “PhraseSets for Evaluating Text Entry Techniques,” Extended Abstracts of CHI2003, pp. 754-755 (2003) (hereinafter “MacKenzie I”). The set contains500 short English phrases with no punctuation, varying from 16-43characters with a high letter frequency correlation with an Englishcorpus. The phrase set was supplemented with news headline phrasesculled from the Internet containing words not found in the testdictionary (e.g., “smsing while driving is risky,” “obama isinaugurated”). Four phrases and one supplemental phrase were randomlyselected for each of sixteen conditions: one for each semaphore(4-choose-1), plus combinations thereof (4-choose-2,4-choose-3, and4-choose-4). These combinations were tested in order to examine if anyof the semaphores conflicted with each other, and to investigate howpeople felt about having a semaphore be absent in another condition. Theorder of the conditions was not counter-balanced. Using the“think-aloud” protocol described in Lewis & Rieman, Task-Centered UserInterface Design: A Practical Introduction (1993) (available viaanonymous ftp at: ftp.cs.colorado.edu), participants were asked in eachof the sixteen conditions to type in the phrases “as quickly aspossible,” but instructed that they should pause and verbalize any newthoughts they had about the semaphores at any moment. During the study,specific questions were asked about each semaphore, and at the end ofthe study, participants were asked to rank-order any semaphores theywould leave on by default.

C. Results

Because this was a usability study aimed at refining the user experienceand design of the semaphores, the visual and auditory parameters of somesemaphores were continually adjusted based on user feedback. As such, itis difficult to accurately interpret raw statistics. However, thenumbers do convey general trends reported here.

1. Unexpected-Key Feedback

Seven of the eleven participants stated that they would leave theunexpected-key feedback semaphore turned on by default, even when usingthe more “distracting” versions shown in FIGS. 7F-7H. Five of the elevenlisted this semaphore as their top choice, stating that they “depended”on it. In particular, they noted how it would alert them to thecandidates area where they would almost certainly find their desiredword as a choice. In fact, many participants claimed that this was their“strategy” for typing as quickly and as accurately as possible.Interestingly, after the settling on a visual design, half theparticipants said they did not perceive the visual cues and only reliedon the auditory cue, and half said that they would turn off the auditorycue, as they considered the visual cue informative enough.

2. Auto-Correction Feedback

All the eleven participants loved the auto-correction semaphore and saidthat they “depended” on knowing when words were being auto-corrected.Furthermore, several participants noted that they could even predictwhen certain words would be auto-corrected (e.g., “obama”). This helpedthem to be prepared to select their replaced word in the candidate area,as shown in, for example, FIG. 9C. Many participants asked to have thereplaced word be automatically added to the dictionary when selectedfrom the candidate area.

3. Key-Trail Feedback

Only five of the eleven participants stated that they would leave thesemaphore on by default, with two of the five claiming that they“depended” on it, and three of the five claiming that it was “cool eyecandy.” Informally, it was observed that those two subjects had veryslow baseline typing speeds. Of the six who did not choose to leave thesemaphore on, two really disliked it (although this may have been due tothe fading rate, which were later adjusted).

4. Key-Prediction Guidance

Only five of the eleven participants stated that they would leave thesemaphore on by default, with three of the five listing it as their topchoice. Of the five, only one claimed he “depended” on it, particularlyfor one-handed use, with the rest stating that they would leave it on“just in case [they] were unsure of how to spell a long word.”

5. Combinations

Initially, two participants disliked the combination of thekey-prediction feedback with the key-trail feedback, but that was whenthe keys were entirely colored. Once the color blue was applied only tothe letters, no complaints were received. No other combinations seemedto bother the participants.

6. Discussion

Given the uniformly positive response for the auto-correction feedback,it was decided not to test this semaphore further. Indeed, for thesubsequent experiment, this semaphore was left on all the time. Thediversity of preferences for the other semaphores suggests that usersshould be given the option of turning specific semaphores on and off ina control panel. With the unexpected-key feedback, some people seemed tobe more tuned to the audio cue and some to the visual, suggesting thatgiving users fine-grained control over which cue is on would bevaluable.

In watching and listening to the usability participants, it was observedthat the semaphores seemed to have more utility for people who typedslower than those who were accustomed to typing quickly on touch screenkeyboards. Interestingly, all of the touch screen phone owners statedthat they would leave the unexpected-key feedback on by default, withtwo of the three listing it as their top choice.

XI. Second Experimental Results

A. Introduction

Given the feedback received from the usability study discussed above, itwas decided to compare the text entry performance of only theunexpected-key feedback and the key-prediction guidance semaphores. Theauto-correction feedback semaphore was incorporated into the baselinefor two reasons: (1) the usability results for auto-correction feedbackwere compelling, and (2) to see how well it would work with the othertwo semaphores for timed text entry. Key-trail feedback was not testedin the second experiment. Although the key-trail feedback receivedusability responses that were as ambivalent as the key-predictionfeedback, key-prediction feedback was tested instead because of relatedprevious research as discussed in MacKenzie & Zhang, “Eye Typing UsingWord and Letter Prediction and a Fixation Algorithm,” Proc. of the ACMSymposium on Eye Tracking Research and Applications—ETRA 2008, pp. 55-58(2008) (hereinafter “MacKenzie II”) and Magnien, et al., “Mobile TextInput with Soft Keyboards: Optimization by Means of Visual Clues,” Proc.of MobileHCI, pp. 337-341 (2004) (hereinafter “Magnien”).

B. Methodology

Eighteen participants (nine males and nine females) between the ages of21 and 39 were recruited using the same professional contracting serviceas in Experiment One. Participants again hailed from a wide variety ofoccupational backgrounds. All participants were compensated for theirtime. Five owned touch screen phones at some time in their life, nineowned QWERTY phones at some time in their life, and seven owned 12-keynumeric phones only (note that these are not exclusive categories).During recruiting, all participants answered that they were familiarwith the QWERTY layout and could type on a normal-size keyboard withoutfrequently looking at the keys.

The MacKenzie and Soukoreff phrase set was again utilized, except inorder to make sure that participants had a chance to hit every letter onthe keyboard, a script to select the shortest sequences of phrases wasused that covered the entire alphabet from A to Z. Supplemental phraseswith words not found in the test dictionary were not included, toattempt to reproduce previous results for the key-prediction guidancesemaphore on a mobile QWERTY touch screen keyboard. For each condition,subjects received eight practice and twenty stimuli items.

All participants were first taught the basics of using the touch screenkeyboard on a mobile device, in particular, a prototype before-marketphone with a 3.5 inch resistive screen having 800×480 WVGA resolution. Atarget phrase was displayed on a desktop computer screen andparticipants were asked to memorize it. They had as much time as theyneeded to memorize the phrase. Participants were asked to memorize thephrases in order to mimic the real experience of entering intended text.When participants felt they were “ready,” their task was to type thephrase into the mobile device “as quickly and as accurately aspossible.” The phrase was left displayed on the computer screen, becausein previous experiments with the same task, some naïve participants (whowere not all university students) experienced difficulties withmemorization under timed conditions. Timing began as soon as theyentered the first letter of the phrase and ended when they hit the“Enter” button twice. The entire experiment took slightly under twohours.

The primary independent variable used was Semaphore, consisting ofBaseline, Unexpected-Key Feedback and Key-Prediction Guidance. Asdependent measures, time to enter text, accuracy, and the efficiencymeasure: keystrokes-per-character or KSPC as described in MacKenzie &Tanaka-Ishii, Text Entry Systems: Mobility, Accessibility, Universality(Morgan Kaufmann Publishers 2007) (hereinafter “MacKenzie III”) wereexamined.

The number of times users pressed the backspace button was used as anadditional measure. Note that users were not allowed to place the cursoronto their typed text for editing since not all participants wereproficient at this task. In short, a simple, within-subject experimentwas conducted where all participants encountered the three Semaphoreconditions in counter-balanced order.

C. Results

1. Time to Enter Text

Given the previously reported success of key-prediction, it washypothesized that the key-prediction guidance semaphore wouldsignificantly reduce the average time participants spent entering ineach phrase. It was also predicted that the unexpected-key feedbacksemaphore would reduce time to enter text as well since severalusability participants had already mentioned using this feedback as astrategy for selecting word prediction choices in the candidates area.Indeed, using repeated measures analysis of variance (ANOVA), asignificant main effect for Semaphore was observed (F_(2,1057)=4.97,p<0.01). However, Tukey post-hoc pair-wise comparisons only revealedsignificant differences between the Baseline and Unexpected-Key Feedback(p<0.01) and between the Unexpected-Key Feedback and the Key-PredictionGuidance (p<0.05), but not between the Baseline and Key-PredictionGuidance. FIG. 16A is a chart 1600 that shows the average time spent perphrase for the Baseline, Key-Prediction Guidance and the Unexpected-KeyFeedback conditions, which were 26.58, 25.81, and 23.95 respectively.

One possible reason for not observing a significant difference in timeto enter text for the Key-Prediction Guidance over the Baseline is thatparticipants were all familiar with the QWERTY layout, unlike thedynamically changing keyboard task described in Magnien. Hence,participants did not experience a need to limit the visual search space.

2. Accuracy

Accuracy was measured in terms of whether or not the participantultimately typed in the correct phrase. Finer-grained measures ofcorrectness were also looked at, such as the Minimal String DistanceError Rate described in MacKenzie III, which computes the distancebetween two strings in terms of the lowest number of edit operations toturn one string into the other. However, these measures could not beused since some of the errors made by users were those related topressing a word prediction or disambiguation candidate, which swaps thetyped text completely. In this case, one keystroke engenders a lot oferrors. Hence, finer-grained measures could be unreliable.

It was hypothesized that both semaphores would result in higher accuracysince the Unexpected-Key Feedback is immediately informing users oferrors and the Key-Prediction Guidance is leading users to the correctspelling. However, in analyzing the data, a main effect for Semaphorewas not observed. All three conditions obtained accuracies between 0.12and 0.14 with fairly wide standard deviations. This deviates fromprevious results described in MacKenzie II in which Key-PredictionGuidance reduced error rates for eye-typing, although it is hard tocompare across modalities.

3. KSPC

It was also hypothesized that the two semaphores would reducekeystrokes-per-character. Indeed, a significant main effect was observedfor Semaphore (F_(2,1057)=6.74, p<0.001), with significant Tukeypost-hoc differences between the Baseline and the Unexpected-KeyFeedback (p<0.05) and between the Baseline and the Key-PredictionGuidance (p<0.05). FIG. 16B shows a chart 1610 displaying the averagekeystrokes-per-character for the Baseline, Key-Prediction Guidance andthe Unexpected-Key Feedback conditions, which were 1.22, 1.10, and 1.09respectively.

Assuming that participants accurately hit only the letter they aresupposed to and not any of the word prediction candidates, the baselineKSPC for a QWERTY layout keyboard is theoretically 1 according toMacKenzie III. Note that all three conditions had a KSPC above 1,indicating that typing on a small 3.5 inch resistive screen with fatfingers is not easy, even with the word prediction and disambiguationcandidates.

4. Number of Backspaces

The number of backspaces was measured as a way to tease apartintentionally corrected mistakes from KSPC, which encompasses that aswell as keystrokes for selecting word prediction candidates. It washypothesized that Unexpected-Key Feedback would reduce the number ofbackspaces by alerting users to a mistake before they enter morecharacters to the end of that mistake, and that Key-Prediction Guidancewould also reduce the number of backspaces by leading users to correctspelling. Indeed, a main effect was observed for Semaphore(F_(2,1057)=7.26, p<0.01) but with just a significant Tukey post-hocdifference between the Baseline and Unexpected-Key Feedback (p<0.001).FIG. 16C is a chart 1620 that shows the average number of backspaces forthe Baseline, Key-Prediction Guidance and the Unexpected-Key Feedbackconditions, which were 3.54, 2.79, and 2.29 respectively.

5. One Versus Two Hands

During the process of running the experiment, an interesting trend wasnoticed; those participants who chose to perform the entire study withtwo thumbs tended to make less use of the word prediction anddisambiguation candidates. Because participants had to maintain the sameway of holding the device throughout the experiment, incoming subjectswere asked thereafter to use either two hands (i.e., two thumbs) or onehand for input. No participant held the device with one hand and typedwith the thumb for the entire experiment, though during practice someparticipants certainly tried out that position. All one-handedparticipants used one hand to hold the device and another to enter textwith their index finger.

While it would have been ideal to run a within-subjects, 3×2 factorialdesign experiment with the OneVsTwoHands added as another independentvariable, because fair amount of data had already been collected,between-subjects analysis of the results was conducted, making sure thatboth one-handed and two-handed users were represented in all possibleorderings of the three conditions studied. Hence, additional statisticalanalyses treating OneVsTwoHands as an additional factor in a univariateANOVA was performed.

For time to enter text, a significant main effect for OneVsTwoHands wasobserved (F_(1,1057)=5.66, p<0.05), as well as a significant interactioneffect with Semaphore (F_(2,1057)=3.68, p<0.05). FIG. 16D is a chart1630 that shows a breakdown of the previous time taken to enter textresult in terms of the number of hands. When using two hands,participants do not seem to be reducing time to enter text in either theKey-Prediction Guidance or the Unexpected-Key Feedback conditions, asshown by the bars 1632, 1633, and 1634 for Baseline, Key-PredictionGuidance, and Unexpected Key Feedback, respectively. However, whenparticipants used one hand, the differences with the Baseline becomemore accentuated, as shown by the bars 1636, 1637, and 1638 forBaseline, Key-Prediction Guidance, and Unexpected Key Feedback,respectively. Although significant main effect for OneVsTwoHands on KSPC(F_(1,1057)=2.77, p<0.01) and number of backspaces (F_(1,1057)=5.46,p<0.05) was observed as well, the same kind of interaction effect withSemaphore was not observed for any dependent variable other than time toenter text. Hence, for KSPC and number of backspaces, the significantpost-hoc differences observed previously recur, regardless of the numberof hands used.

6. Final Questionnaire

After the experiment, users were asked to pick their favorite conditionand to rank-order which of the two semaphores they would leave on bydefault. This was done to confirm qualitative findings from theusability study. Thirteen of the eighteen participants found theUnexpected-Key Feedback condition to be the most favorable. No onepicked the Baseline. With respect to rank-ordering, eight of theeighteen participants said they would leave on the key-predictionfeedback semaphore and nine of the eighteen said they would leave on theunexpected-key feedback semaphore. The fact that only nine, and notthirteen, participants said they would leave it on implies that althoughsome people found the semaphore useful, it is disruptive enough to themthat they would turn it off by default and turn it on as needed.

D. Discussion and Design Implications

From conducting the usability study, it was observed that visual cuesfor semaphores cannot grab too much attention. If they do, usersgenerally dislike them. Toning down the semaphores allows users who wantto use the visual cues, to mentally grab hold of them, and those who donot, to ignore them. As such, it is desirable that semaphores bedesigned to be very visually subtle.

Thus, these experimental results indicate that both the unexpected-keysemaphore and the key-prediction guidance semaphore have the ability toimprove text entry performance.

XIII. Example Alternatives and Combinations

Any of the methods described herein can be performed via one or morecomputer-readable media (e.g., storage or other tangible media)comprising (e.g., having or storing) computer-executable instructionsfor performing (e.g., causing a computing device to perform) suchmethods. Operation can be fully automatic, semi-automatic, or involvemanual intervention.

Having described and illustrated the principles of our innovations inthe detailed description and accompanying drawings, it will berecognized that the various embodiments can be modified in arrangementand detail without departing from such principles. It should beunderstood that the programs, processes, or methods described herein arenot related or limited to any particular type of computing environment,unless indicated otherwise. Various types of general purpose orspecialized computing environments may be used with or performoperations in accordance with the teachings described herein. Elementsof embodiments shown in software may be implemented in hardware and viceversa.

In view of the many possible embodiments to which the principles of ourinvention may be applied, we claim as our invention all such embodimentsas may come within the scope of the following claims and equivalentsthereto.

1. A method, comprising: receiving first input data comprising one or more input words from a keyboard; providing one or more feedback semaphores, wherein the feedback semaphores are operable to alert a user using the keyboard that one or more suggestion candidates are available; automatically displaying the one or more suggestion candidates, wherein the suggestion candidates are each associated with an alternative word for one or more of the input words; automatically receiving a touch screen input selecting one of the suggestion candidates; and modifying the first input data using the alternative word associated with the selected suggestion candidate.
 2. The method of claim 1, further comprising: based on the first input data, generating one or more alternative words for one or more of the input words; auto-correcting the first input data by automatically replacing the input word with one of the alternative words; and wherein the providing the feedback semaphores occurs immediately after the auto-correcting.
 3. The method of claim 2, wherein: the first input data comprises a character designated as a delimiter for the input word; and the providing the feedback semaphores occurs based on receiving the delimiter character in the first input data and on word probability data associated with the input word.
 4. The method of claim 1, wherein the feedback semaphores include one or more of the following: playing a sound, generating haptic feedback, highlighting one or more keys on the keyboard, highlighting a background area of the keyboard, and highlighting a suggestion candidates area.
 5. The method of claim 1, wherein: the feedback semaphores include highlighting one or more delimiter keys on the keyboard; and the receiving a single touch screen input comprises receiving a key press for one of the highlighted delimiter keys.
 6. The method of claim 1, wherein the feedback semaphores are not provided until the input word is designated incorrect using a dictionary, a common speller application programming interface, or an input history data source.
 7. The method of claim 6, wherein: the first input data comprises a character designated as a delimiter for the input word; and the providing the feedback semaphores occurs based on receiving the delimiter character in the first input data.
 8. The method of claim 1, wherein the providing the feedback semaphores does not occur until a delimiter key for the input word is received in the first input data.
 9. The method of claim 1, wherein: the keyboard is a touch screen keyboard; and the receiving text input data further comprises playing a randomly selected keypress sound for a character of the first input data, wherein the keypress sound is selected from a group including at least two or more keypress sounds.
 10. The method of claim 1, further comprising: receiving typing speed data for at least a portion of the first input data; and based on the typing speed data, selecting one or more feedback semaphores from a group including at least two or more of the following: a tooltip balloon, audio feedback, haptic feedback, highlighting one or more delimiter keys, highlighting the keyboard, and wherein the one or more feedback semaphores are the selected feedback semaphores.
 11. The method of claim 1, further comprising: detecting whether the user is using the keyboard with one hand or two hands, and; based on the detecting, selecting one or more feedback semaphores from a group including at least two or more of the following: a tooltip balloon, audio feedback, haptic feedback, highlighting one or more delimiter keys, highlighting the keyboard, and wherein the one or more feedback semaphores are the selected feedback semaphores.
 12. A computer-readable storage media storing computer-readable instructions that when executed by a computer cause the computer to perform the method of claim
 1. 13. A computer-readable storage media storing computer-readable instruction that when executed by a computer cause the computer to perform a method, the method comprising: receiving text input data comprising at least one word using a keyboard coupled to a touch screen; detecting that an event has occurred based on the text input data and based on the detected event, automatically displaying on the touch screen one or more suggestion candidates for the at least one word based on the text input data and one or more candidate sources, wherein each of the candidates is associated with at least one next word designated as likely to follow the at least one word; receiving a single touch screen input selecting one of the suggestion candidates; and based on the single touch screen input, automatically modifying the text input data by adding the at least one next word associated with the selected suggestion candidate to the text input data.
 14. The computer-readable storage media of claim 13, wherein the event is detected based on a measured keystroke latency exceeding a threshold value.
 15. The computer-readable storage media of claim 13, wherein the event is an auto-correction of the text input data or receiving a delimiter key in the text input data.
 16. The computer-readable storage media of claim 13, further comprising automatically displaying on the touch screen one or more replacement suggestion candidates for the at least one word based on the text input data and one or more candidate sources, wherein each of the replacement suggestion candidates is associated with at least one replacement word for the at least one word; and wherein the displaying includes displaying the replacement suggestion candidates in a manner that distinguishes the replacement suggestion candidates from the suggestion candidates.
 17. The computer-readable storage media of claim 13, further comprising: immediately after the automatically adding the at least one word, displaying one or more suggestion candidates for the text input data, which includes the selected next word; and repeating the automatically modifying the text input data.
 18. A mobile device, comprising: one or more processing units operable to execute computer-executable instructions for text entry and correction; one or more memory units coupled to the processing units; one or more touch screens coupled to the mobile device configurable to have a text display area, a suggestion candidates area, and a touch screen keyboard area, wherein the text display area, the suggestion candidates area, and the touch screen keyboard area occupy distinct, non-overlapping areas of the touch screens, and wherein the one or more touch screens are operable to receive touch input over at least a portion of the touch screen keyboard area and the suggestion candidates area; storage for storing the computer-executable instructions for text entry and correction using: a text input module for receiving text input using the touch screen keyboard and displaying at least a portion of the text input in the text display area; a candidate generation module for generating one or more suggestion candidates comprising one or more replacement suggestions and/or one or more next word suggestions for an input word of the text input; an indicator generation module for producing one or more indicators that can notify a user that the suggestion candidates are available; a suggestion presentation module for presenting the suggestion candidates associated with the input word in the suggestion candidates area and receiving touch screen user input for selecting one of the presented suggestions, wherein: if the selected suggestion is a replacement suggestion, replacing the input word with the selected suggestion; and if the selected suggestion is a next word suggestion, adding the selected suggestion as a next word subsequent to the input word in the text input.
 19. The mobile device of claim 18, wherein the indicator generation module and the suggestion presentation module are configured to not produce indicators and to not present suggestion candidates, respectively, until one of the input words is designated incorrect by a dictionary, a common speller application programming interface, or an input history data source.
 20. The mobile device of claim 18, wherein the indicator generation module and the suggestion presentation module are configured to not produce indicators and to not present suggestion candidates, respectively, until a delimiter character is received immediately after the input word. 